Systems and methods for interfaces featuring surface-based haptic effects

ABSTRACT

Systems and methods for interfaces featuring surface-based haptic effects are described. One described system includes a sensor configured to detect a touch in a touch area when an object contacts a touch surface. The touch surface may correspond to the display area or may correspond to a non-display surface of a computing device or peripheral interfaced to a computing device. The system can further include an actuator in communication with the processor and coupled to the touch surface, the actuator configured to output a haptic effect and a processor configured to select a haptic effect to generate. The haptic effect can be selected based on a position of the touch and recognizing an input gesture provided by the touch and/or content displayed in a graphical user interface at a location mapped to a position in the touch area at or near the touch. The haptic effect may provide a texture, simulate an obstacle, and/or adjust the coefficient of friction of the surface.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is a continuation of and claims the benefit ofapplication Ser. No. 15/988,359, filed on May 24, 2018, and entitled“Systems and Methods for Interfaces Features Surface-Based HapticEffects,” which is a continuation of and claims the benefit ofapplication Ser. No. 12/696,908, filed on Jan. 29, 2010, and entitled“Systems and Methods for Interfaces Features Surface-Based HapticEffects,” now U.S. Pat. No. 10,007,340, issued on Jun. 26, 2018, whichclaims the benefit of U.S. Provisional Patent Application No.61/262,038, entitled “Friction Rotary Device for Haptic Feedback” filedNov. 17, 2009, and claims the benefit of U.S. Provisional PatentApplication No. 61/262,041, entitled “System and Method for IncreasingHaptic Bandwidth in an Electronic Device” filed Nov. 17, 2009, andclaims the benefit of U.S. Provisional Patent Application No.61/159,482, entitled “Locating Features Using a Friction Display,” filedMar. 12, 2009, the entirety of all of which is hereby incorporated byreference herein in their entirety.

Application Ser. No. 12/696,908 is related to U.S. patent applicationSer. No. 12/697,010, filed the same day and entitled “Systems andMethods for a Texture Engine,” (Attorney Docket No. IMM354(51851-383720)), which is incorporated by reference herein in itsentirety.

Application Ser. No. 12/696,908 is related to U.S. patent applicationSer. No. 12/697,042, filed the same day and entitled “Systems andMethods for Using Multiple Actuators to Realize Textures,” (AttorneyDocket No. IMM355 (51851-383719)), which is incorporated by referenceherein in its entirety.

Application Ser. No. 12/696,908 is related to U.S. patent applicationSer. No. 12/697,037, filed the same day and entitled “Systems andMethods for Using Textures in Graphical User Interface Widgets,”(Attorney Docket No. IMM356 (51851-383718)), which is incorporated byreference herein in its entirety.

Application Ser. No. 12/696,908 is related to U.S. patent applicationSer. No. 12/696,893, filed the same day and entitled “Systems andMethods for Providing Features in a Friction Display,” (Attorney DocketNo. IMM357 (51851-383714)), which is incorporated by reference herein inits entirety.

Application Ser. No. 12/696,908 is related to U.S. patent applicationSer. No. 12/696,900, filed the same day and entitled “Systems andMethods for Friction Displays and Additional Haptic Effects,” (AttorneyDocket No. IMM358 (51851-383716)), which is incorporated by referenceherein in its entirety.

BACKGROUND

Touch-enabled devices have been increasingly popular. For instance,mobile and other devices may be configured with touch-sensitive displaysso that a user can provide input by touching portions of thetouch-sensitive display. As another example, a touch-enabled surfaceseparate from a display may be used for input, such as a trackpad,mouse, or other device.

For example, a user may touch a portion of the display or surface thatis mapped to an on-screen graphical user interface (GUI), such as abutton or control. As another example, a gesture may be provided, suchas a sequence of one or more touches, drags across the surface, or otherrecognizable patterns sensed by the device. Although touch-enableddisplays and other touch-based interfaces have greatly enhanced devicefunctionality, drawbacks remain. For instance, even if a keyboard isdisplayed on a screen, a user accustomed to a physical keyboard may nothave the same experience while using the touch-enabled device.

SUMMARY

Embodiments of the present invention can provide a touch-enabled devicefeaturing surface-based haptic effects including, but not limited to,changes in texture, changes in a coefficient of friction of the touchsurface, and/or simulation of boundaries or obstacles on the touchsurface. Through the use of such features, devices may be more userfriendly and may provide an overall more compelling user experience.

In one embodiment, a system comprises a sensor configured to detect atouch in a touch area when an object contacts a touch surface, the toucharea mapped to a display area of a graphical user interface. The touchsurface may correspond to the display area or may correspond to anon-display surface of a computing device or peripheral interfaced to acomputing device.

The system can further include an actuator in communication with theprocessor and coupled to the touch surface, the actuator configured tooutput a haptic effect. The system can include one or more processors incommunication with actuator and sensor, the processor configured toselect a haptic effect to generate based on identifying a position ofthe touch and at least one of (a) recognizing an input gesture providedby the touch or (b) content displayed in the graphical user interface ata location in the display area mapped to a position in the touch area ator near the touch. The processor can command the actuator to generatethe selected haptic effect while the touch is occurring to therebyprovide a texture, simulate an obstacle or boundary, and/or provide anadjusted coefficient of friction of the display. These effects may beachieved using one or more actuators to provide haptic effects(including, but not limited to, vibrotactile effects) and/or usingactuators that change the coefficient of friction of the display in acontrolled manner.

These illustrative embodiments are mentioned not to limit or define thelimits of the present subject matter, but to provide examples to aidunderstanding thereof Illustrative embodiments are discussed in theDetailed Description, and further description is provided there.Advantages offered by various embodiments may be further understood byexamining this specification and/or by practicing one or moreembodiments of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

A full and enabling disclosure is set forth more particularly in theremainder of the specification. The specification makes reference to thefollowing appended figures.

FIG. 1A shows an illustrative system for providing surface-based hapticeffects.

FIG. 1B shows an external view of one embodiment of the system shown inFIG. 1A.

FIG. 1C illustrates another embodiment of the system shown in FIG. 1A.

FIG. 2 is a diagram illustrating an array of pixels 200 which can beused in some embodiments to select a haptic effect.

FIG. 3 is a flowchart showing an illustrative method for providing aninterface with surface-based haptic effects

FIG. 4A is a diagram illustrating an external view of a systemcomprising a computing device that features a touch-enabled display.

FIG. 4B shows a cross-sectional view of the device of FIG. 4A.

FIGS. 4C-4D provides an example of a graphical user interface in thedevice of FIG. 4A as the processor is configured to select a hapticeffect in response to recognizing a page turn input gesture.

FIGS. 5A-5B each illustrate a system comprising a computing devicefeaturing a touch surface and interfaced to a separate display.

DETAILED DESCRIPTION

Reference will now be made in detail to various and alternativeillustrative embodiments and to the accompanying drawings. Each exampleis provided by way of explanation, and not as a limitation. It will beapparent to those skilled in the art that modifications and variationscan be made. For instance, features illustrated or described as part ofone embodiment may be used on another embodiment to yield a stillfurther embodiment. Thus, it is intended that this disclosure includemodifications and variations as come within the scope of the appendedclaims and their equivalents.

Illustrative Example of a Device Using a Variable Friction Interface

One illustrative embodiment of the present invention comprises acomputing system such as an iPod® portable music device or iPhone®mobile device, both available from Apple Inc. of Cupertino, Calif., or aZune(R) portable device, available from Microsoft Corporation ofRedmond, Wash. The computing system can include and/or may be incommunication with one or more sensors, such as an accelerometer, aswell as sensors (e.g., optical, resistive, or capacitive) fordetermining a location of a touch relative to a display areacorresponding in this example to the screen of the device.

As the user interacts with the device, one or more actuators are used toprovide tactile effects. For example, as a user moves a finger acrossthe device, the coefficient of friction of the screen can be variedbased on the position, velocity, and/or acceleration of the finger.Depending on how the friction is varied, the user may perceive a featureand/or a texture. As a particular example, the friction may be varied sothat the user perceives a bump, border, or other obstacle correspondingto an edge of an on-screen button. As will be discussed in furtherdetail below, varying the coefficient of friction can be used in anynumber of ways to provide feedback to a user.

Illustrative Systems for Providing Surface-Based Haptic Effects

FIG. 1A shows an illustrative system 100 for providing a surface-basedhaptic effect. Particularly, in this example, system 100 comprises acomputing device 101 featuring a processor 102 interfaced with otherhardware via bus 106. A memory 104, which can comprise any suitabletangible (and non-transitory) computer-readable medium such as RAM, ROM,EEPROM, or the like, embodies program components that configureoperation of the computing device. In this example, computing device 101further includes one or more network interface devices 110, input/output(I/O) interface components 112, and additional storage 114.

Network device(s) 110 can represent any components that facilitate anetwork connection. Examples include, but are not limited to, wiredinterfaces such as Ethernet, USB, IEEE 1394, and/or wireless interfacessuch as IEEE 802.11, Bluetooth, or radio interfaces for accessingcellular telephone networks (e.g., transceiver/antenna for accessing aCDMA, GSM, UMTS, or other mobile communications network).

I/O components 112 may be used to facilitate connection to devices suchas a one or more displays, keyboards, mice, speakers, microphones,and/or other hardware used to input data or output data. Storage 114represents nonvolatile storage such as magnetic, optical, or otherstorage media included in device 101.

System 100 further includes a touch surface 116, which is in thisexample integrated into device 101. Touch surface 116 represents anysurface that is configured to sense tactile input of a user. One or moresensors 108 are configured to detect a touch in a touch area when anobject contacts a touch surface and provide appropriate data for use byprocessor 102. Any suitable number, type, or arrangement of sensors canbe used. For example, resistive and/or capacitive sensors may beembedded in touch surface 116 and used to determine the location of atouch and other information, such as pressure. As another example,optical sensors may be used to determine the touch position.

In this example, an actuator 118 in communication with processor 102 iscoupled to touch surface 116. In some embodiments, actuator 118 isconfigured to output a haptic effect varying a coefficient of frictionof the touch surface in response to a haptic signal. Additionally oralternatively, actuator 118 may provide haptic effects that move thetouch surface in a controlled manner. Some haptic effects may utilize anactuator coupled to a housing of the device, and some haptic effects mayuse multiple actuators in sequence and/or in concert. For example, thecoefficient of friction can be varied by vibrating the surface atdifferent frequencies. Different combinations/sequences of variance canbe used to simulate the feeling of a texture.

Although a single actuator 118 is shown here, embodiments may usemultiple actuators of the same or different type to vary the coefficientof friction of the touch surface. For example, a piezoelectric actuatoris used in some embodiments to displace some or all of touch surface 116vertically and/or horizontally at ultrasonic frequencies. In someembodiments, multiple actuators such as eccentric rotating mass motorsand linear resonant actuators can be used alone or in concert to providedifferent textures, friction variances, and other haptic effects. Otherexamples of actuators include electroactive polymers, shape memoryalloys, electrostatic, and magnetostrictive actuators.

Turning to memory 104, exemplary program components 124, 126, and 128are depicted to illustrate how a device can be configured in someembodiments to provide a variable-friction display. In this example, adetection module 124 configures processor 102 to monitor touch surface116 via sensor(s) 108 to determine a position of a touch. For example,module 124 may sample sensor 108 in order to track the presence orabsence of a touch and, if a touch is present, to track the location,path, velocity, acceleration, pressure and/or other characteristics ofthe touch over time.

Haptic effect determination module 126 represents a program componentthat analyzes data regarding touch characteristics to select a hapticeffect to generate. For example, in some embodiments, an input gesturecomprising a sequence of one or more touches may be recognized andcorrelated to one or more haptic effects. As another example, some orall of the area of touch surface 116 may be mapped to a graphical userinterface. Different haptic effects may be selected based on thelocation of a touch in order to simulate the presence of a feature byvarying the friction of touch surface 116 so that the feature is “felt”when a corresponding representation of the feature is seen in theinterface. However, haptic effects may be provided via touch surface 116even if a corresponding element is not displayed in the interface (e.g.,a haptic effect may be provided if a boundary in the interface iscrossed, even if the boundary is not displayed).

Haptic effect generation module 128 represents programming that causesprocessor 102 to generate and transmit a haptic signal to actuator(s)118 to generate the selected haptic effect at least when a touch isoccurring. For example, generation module 128 may access storedwaveforms or commands to send to actuator 118. As another example,haptic effect generation module 128 may receive a desired coefficient offriction and utilize signal processing algorithms to generate anappropriate signal to send to actuator(s) 118. As a further example, adesired texture may be indicated along with target coordinates for thetexture and an appropriate waveform sent to one or more vibrotactileactuators to generate appropriate displacement of the surface (and/orother device components) to provide the texture.

A touch surface may or may not overlay (or otherwise correspond to) adisplay, depending on the particular configuration of a computingsystem. In FIG. 1B, an external view of a computing system 100B isshown. Computing device 101 includes a touch-enabled display 116 thatcombines a touch surface and a display of the device. The touch surfacemay correspond to the display exterior or one or more layers of materialabove the actual display components.

In this example, a haptic effect is selected based on the content oftext displayed in a graphical user interface 130. Particularly, asindicated by interface element 132, a “find” command has been used totrigger a search query. A haptic effect can be selected to generate ahaptic effect at positions in the touch area mapped to positions in theuser interface corresponding to the searched content. In this example,the touch surface is mapped directly to the display area, and so whenthe searched content (the word “patent”) is located, it is highlightedin interface 130 as shown at 134, 136, and 138.

In accordance with aspects of the present subject matter, theprocessor(s) of device 101 can select an appropriate haptic effect sothat when a user touches screen 116 at or near 134, 136, and/or 138, thecoefficient of friction of the screen is varied. For instance, this mayallow a user to “skim” the text using his or her fingers, with the textof interest tactilely highlighted by having a higher (or lower)coefficient of friction or a texture distinct from other displayedcontent. More complex effects may also be created; for instance, aborder or texture around the highlighted words can be generated usingone or more actuators of device 101.

As was noted above, a touch surface need not overlay a display. FIG. 1Cillustrates another example of a touch-enabled computing system 100C. Inthis example, a computing device 101 features a touch surface 116 whichis mapped to a graphical user interface provided in a display 122 thatis included in computing system 120 interfaced to device 101. Forexample, computing device 101 may comprise a mouse, trackpad, or otherdevice, while system 120 may comprise a desktop or laptop computer,set-top box (e.g., DVD player, DVR, cable television box), or anothercomputing system. As another example, touch surface 116 and display 122may be included in the same device, such as a touch-enabled trackpad ina laptop computer featuring display 122.

Whether integrated with a display or otherwise, the depiction of 2-Drectangular touch surfaces in the examples herein is not meant to belimiting. Other embodiments include curved or irregular touch-enabledsurfaces that are further configured to provide surface-based hapticeffects.

Returning to FIG. 1C, in this example the user interface featurestextual content (the word “Examination”) in the process of beingselected in response to a user gesture input via touch surface 116.Particularly, first portion 140 is highlighted based on movement offinger 146 through area 150 of touch surface 116. As the word isselected, finger 146 will move to the position shown at 148, while inthe graphical user interface, portions 142 and 144 of the word arehighlighted.

A haptic effect can be provided in response to the gesture and/or basedon the content of the word. For instance, as indicated by the differentcross-hatching of areas 150, 152, and 154, the coefficient of frictionof surface 116 can be changed as the word is highlighted. In someembodiments, content of selected text or other items is used todetermine the haptic effect.

For instance, the friction may vary based on a “height map” of theselected text, with short letters (“e”, “r”) having a low height andtaller letters (such as “l”, “f”) having a higher height. A frequencytransform can be used to find the frequency content of the word;alternatively, a distance between the letter could be used to determinea “feel” for the word. As another example, the friction may be adjustedbased simply on the selection, such as an increase in friction as theend of a word, phrase, or other unit is reached; this may help the useravoid selecting undesired portions of adjacent content. Boundariesbetween words can, of course, be recognized by identifying spaces, whilegrammar analysis (e.g., analyzing context, usage) as is known in the artcan be used to recognize phrases.

In this example, textual selection was described based on left-to-righthighlighting. Surface-based textual effects may be associated with othertypes of selection actions. For example, a user may draw a box bytouching at an origin point and then dragging outward with a secondobject, with the two points representing opposite corners of a boundingbox. A haptic effect such as a varying friction or texture can beprovided while the selection is in progress and/or after selection iscompleted. Additionally or alternatively, while the area is selected,the portion of the touch area corresponding to the selected area mayhave a distinct texture or coefficient of friction. As another example,the selected area may be indicated haptically by a border or perimeter(e.g., a haptic click, pop, or friction variance to identify when atouch crosses into or out of the selected area).

Illustrative Methods for Determining Haptic Effects to Provide

FIG. 2 is a diagram illustrating an array of pixels 200 which can beused in some embodiments to select a haptic effect to provide based onmapping a detected touch to one or more pixel locations. Array 200 may,for instance, comprise a bitmap or other image file having one or morevalues associated with each pixel, with the value(s) used to determinehow haptic effects are to be provided. This example shows a relativelysmall number of pixels; in practice, the array may comprise thousands ormillions of pixels.

In some embodiments, a position of a touch within a touch area definedby the touch surface can be returned and then converted to one or morepixel addresses. The value or values associated with the pixeladdress(es) can be accessed and used to drive the actuator(s) of thehaptic device (including the variable friction device or devices). Forinstance, each pixel address may be associated with an intensity valuethat is correlated to an amplitude with which a piezoelectric actuatoris to be driven. As a more complex example, each pixel address may beassociated with three intensity values (i.e., RGB). Each of the threeintensity values can be associated with a different actuator intensityin some embodiments. As another example, some values may specifyintensity and others specify duration of operation. As a furtherexample, different pixel intensity values may be correlated to differentdesired textures or components used to drive actuators to simulate asingle texture. Still further, a multilayer RGB image file may be used,with each layer corresponding to a particular actuator.

In this example, a first plurality of pixels 202 are associated withintensity values and form a “block,” while a second plurality of pixels204 form a different pattern. The different patterns may result indifferent effects based on how an object encounters the patterns. Forinstance, an object moving from top to bottom may encounter a firstchange in friction caused by pattern 202 and then a different variancecaused by pattern 204; the same patterns encountered in reverse order(i.e., bottom to top) may simulate a different feel entirely.

The mapping between the touch surface and graphical user interface maybe absolute or may be scaled. For example, in some embodiments, a touchlocation is directly correlated to a corresponding pixel address (orpixel addresses), with the associated values used in selecting hapticeffects to drive the actuator(s). In other embodiments, the touchlocation and velocity is considered. For example, the actual touchaddress may be above pattern 202 but may be associated with a vectorindicating motion towards pattern 202, and thus the values of pixels inpattern 202 are used to drive the actuators in time for the hapticeffect to be played appropriately.

FIG. 3 is a flowchart showing an illustrative method 300 for providingan interface with surface-based haptic effects. Block 302 representsdetermining a position of a touch in a touch area. For example, aprocessor may utilize one or more sensors embedded in or viewing atouch-enabled display or surface to track a position of a touch on thesurface. Based on the current and/or past position of the touch, aninteraction with a graphical user interface mapped to the touch area canbe determined. Based on the interaction, one or more haptic effects canbe selected, such as varying the friction of the touch surface and/orother effects.

In this example, the interaction is recognized at block 304 and 312,where the method determines first, at 304, whether a gesture isindicated. A gesture can be recognized as a sequence of one or moretouches or patterns of touch, such as based on a direction and length ofa swipe across the screen, a sequence of discrete touches in a pattern,or another recognizable interaction. In this example, if a gesture isrecognized, flow moves to block 306, which represents determining adesired haptic effect associated with the gesture.

For example, a “Z”-shaped touch trajectory may be recognized as a typeof input gesture based on pattern recognition carried out by a processorof the device while the gesture is in progress. One or more hapticeffects may be associated with the “Z”-gesture in data accessible to theprocessor indicating an effect to output while the gesture is inprogress and/or after the gesture is complete. For example, the data mayprovide for the surface to take on a texture or a change in friction asthe gesture nears completion. Additionally or alternatively, a textureor coefficient of friction of the display may change after the gestureis recognized in order to confirm input of the gesture.

If at block 304 a gesture is not recognized, flow moves to block 312, inwhich the method evaluates whether a GUI-based feature is present at ornear the position of the touch. As discussed herein, various featurescan be present in a graphical user interface mapped to the touchsurface, and the features can be associated with haptic effects. Thefeatures may or may not correspond to content actually displayed in thegraphical user interface. Block 312 represents determining if one ormore of such features are at the touched location or near the touchedlocation and block 314 represents determining one or more haptic effectscorresponding to the feature.

For example, the current pixel location and/or a projected pixellocation for the touch based on a velocity of the touch can be comparedto a bitmap specifying haptic effects for various pixel positions. Basedon the haptic effect(s), suitable haptic signals can beaccessed/generated to provide the output specified in the bitmap.

As another example, a current or projected location of a touch can becompared to data identifying the location of GUI features such ascontrols, textual content, boundaries, and the like. Then, if a GUIfeature is identified at the location, data associating one or morehaptic effects to the feature can be accessed. For instance, a processormay track the location of a touch and determine the touch is at orapproaching a position in the touch area mapped to a particular control(e.g., a button) in the graphical user interface. The processor can thenconsult a listing of interface elements to determine a haptic effect(e.g., a texture, a friction variation) associated with the button and,based on the haptic effect, take further actions to generate the hapticeffect.

In this example, both blocks 306 and 314 lead to block 308, whichrepresents accessing or generating one or more haptic signals togenerate the selected haptic effect(s). For example, a processor mayaccess drive signals stored in memory and associated with particularhaptic effects. As another example, a signal may be generated byaccessing a stored algorithm and inputting parameters associated with aneffect. For example, an algorithm may output data for use in generatinga drive signal based on amplitude and frequency parameters. As anotherexample, a haptic signal may comprise data sent to an actuator to bedecoded by the actuator. For instance, the actuator may itself respondto commands specifying parameters such as amplitude and frequency.

Block 310 represents transmitting the haptic signal to the actuator(s)to generate the desired effect(s). For instance, if an analog drivesignal is to be provided, a processor can utilize an onboard D/Aconverter to create the signal. If a digital command is provided to theactuator, an appropriate message can be generated by an I/O bus of theprocessor. The haptic effect may be felt at the point of the touchand/or elsewhere. For example, if a two-finger input gesture isprovided, the texture/coefficient of friction at the first finger may bechanged in response to recognizing movement of the second finger.

In some embodiments, a baseline haptic signal may be sent to theactuator(s) to generate an ambient haptic effect even in the absence ofa selected haptic effect in order to enhance the range of potentialeffects the device can produce. Thus, transmitting a haptic signal maycomprise sending a “stop” command, a “zero” or minimal signal, oranother signal to the actuator to reduce intensity as appropriate.

As an example, use of certain actuators, such as piezoelectricactuators, may allow for reduction in the coefficient of friction of atouch surface but not an increase in the coefficient of friction. Toprovide a range of options, a baseline signal may be provided so thatthe “ordinary” friction level of the touch surface is below thecoefficient of friction the touch surface would have when static.Accordingly, haptic effects may be defined with respect to the baseline,rather than static, value. If maximum friction is desired, a “zero”signal may be sent to the piezoelectric actuator to stop movement of thesurface.

Surface-based haptic effects may take any suitable form. For example,some haptic effects may comprise variations in the friction of the touchsurface—some portions may be rendered “slicker” or “rougher” thanothers. As another example, vibrotactile effects may be used, such asvibrations or series of vibrations. Vibrotactile effects and/orvariations in friction may be used to simulate the feeling of distinctfeatures, such as boundaries or obstacles. For example, a boundary oredge may be simulated by an increase in friction, with the frictiondecreasing if the boundary is crossed (in some instances).

Vibrotactile effects and/or variations in friction may additionally oralternatively be used to simulate various textures. Additional detailregarding generation and use of textures can be found in U.S. patentapplication Ser. Nos. 12/697,010, 12/697,042, and 12/697,037, referencedabove and entitled “Systems and Methods for a Texture Engine,” (AttorneyDocket IMM354 (51851-383720)), “Systems and Methods for Using MultipleActuators to Realize Textures,” (Attorney Docket No. IMM355(51851-383719)), and “Systems and Methods for Using Textures inGraphical User Interface Widgets,” (Attorney Docket No. IMM356(51851-383718)), respectively. For instance, patterns of differingfriction or patterns of vibration may be provided to mimic the feelingof textures such as brick, rocks, sand, grass, fur, various fabrictypes, water, molasses, and other fluids, leather, wood, ice, lizardskin, metals, and other texture patterns. Other textures not analogousto real-world textures may also be used, such as high-magnitudevibrotactile or other feedback when a “danger” texture is desired.

Although in this example blocks 304 and 312 were considered in thealternative, in some embodiments a method can consider whether a gestureis in progress while also evaluating whether the touch is occurring at alocation mapped to a portion of a GUI having one or more tactilefeatures.

Additional Illustrative Embodiments of Interface Behavior

FIG. 4A is a diagram illustrating an external view of a system 400comprising a computing device 401 that features a touch-enabled display402. FIG. 4B shows a cross-sectional view of device 401. Device 401 maybe configured similarly to device 101 of FIG. 1A, though components suchas the processor, memory, sensors, and the like are not shown in thisview for purposes of clarity.

As can be seen in FIG. 4B, device 401 features a plurality of actuators418 and an additional actuator 422. Actuator 418-1 may comprise anactuator configured to impart vertical force to display 402, while 418-2may move display 402 laterally. In this example, the actuators arecoupled directly to the display, but it should be understood that theactuators could be coupled to another touch surface, such as a layer ofmaterial on top of display 402. Additional actuator 422 may be coupledto a housing containing the components of device 401. In the examples ofFIGS. 4A-4D, the area of display 402 corresponds to the touch area,though the principles could be applied to a touch surface completelyseparate from the display.

In one embodiment, actuators 418 each comprise a piezoelectric actuator,while additional actuator 422 comprises an eccentric rotating massmotor, a linear resonant actuator, or another piezoelectric actuator.Actuator 422 can be configured to provide a vibrotactile haptic effectin response to a haptic signal from the processor. The vibrotactilehaptic effect can be utilized in conjunction with surface-based hapticeffects and/or for other purposes.

In some embodiments, either or both actuators 418-1 and 418-2 cancomprise an actuator other than a piezoelectric actuator. Additionally,a single actuator 422 is shown, although multiple other actuators can becoupled to the housing of device 401 and/or other actuators 422 may becoupled elsewhere. Device 401 may feature multiple actuators 418-1/418-2coupled to the touch surface at different locations, as well.

Turning back to FIG. 4A, a graphical user interface is shown at 420 toinclude a generalized feature 430. In this example, a haptic effect isselected for output based on the position of a touch represented bymovement of finger 426 downward to position 428. Particularly, as can beseen in FIG. 4B, actuators 418-1, 418-2, and/or 422 are provided withappropriate haptic signals to provide surface-based haptic feedback asindicated at 432, 434, and 436. The different cross-hatching is intendedto represent different “feel” of the touch surface due to the actuators.For instance, 432, 434, and 436 can represent variations in the textureor coefficient of friction of the touch surface that generate thedesired haptic effect. In one embodiment, the feeling of a box can besimulated by having a first area 432 of higher friction followed by asecond area 434 of lower friction and a third area 436 of higherfriction.

FIGS. 4C-4D provides an example of a graphical user interface 424 as theprocessor is configured to select a haptic effect in response torecognizing a page turn input gesture, with the haptic effect selectedto vary the touch surface in the direction of the page turn inputgesture as the gesture is provided. FIG. 4C shows a first interfacestate 424-1 in which a finger 438 touches an onscreen representation ofa stack of pages 440.

As shown at 442 in FIG. 4D, the finger has provided a right-to-leftinput gesture. In this example, software of computing device 401configures the processor to recognize a right-to-left trajectorybeginning from a page corner as a page turn input gesture. Thus, theinterface moves to state 424-2 where the page corner is lifted and thenext page is visible at 444. Additionally, as the gesture is in progress(and/or after the gesture is recognized), a surface-based haptic effectcan be provided. For example, as finger 438 moves from right to left,the coefficient of friction of the touch surface can be varied (e.g., byincreasing) to simulate the feel of a page turn. As another example, aseries of friction changes can be provided, or one or more texturechanges may occur.

FIGS. 5A-5B each illustrate a system 500 comprising a computing device501 featuring a touch surface 502. Computing device 501 can beconfigured similarly to devices 401 and 101 of FIGS. 4 and 1. In theseexamples, touch surface 502 is separate from a display that features agraphical user interface mapped to the touch area. A separate display504 is shown in FIGS. 5A-5B and interfaced to device 501.

For example, device 501 may represent a computing device interfaced todisplay 504, such as a laptop computer with a built-in trackpadcorresponding to the touch surface. As another example, computing device501 may itself be a peripheral device including the touch surface 502,actuators, and suitable processing circuitry commanded by a computingdevice to which device 501 is interfaced. In any event, it will beunderstood that the principles discussed in FIGS. 5A-5B are equallyapplicable to embodiments in which the touch surface corresponds to thedisplay or a material above the display. Additionally, the principlescan, of course, be used when a touch surface is separate from a display(e.g., a touch surface on a mouse, computer trackpad, etc.).

Turning to FIG. 5A, in this example the processor of device 501 isconfigured to select a haptic effect in response to recognizing agesture indicating movement of an on-screen content item from a firstlocation to a second location. Graphical user interface 506 includeswindows 508 and 510. Element 512 may represent a file, folder, or otherelement that can be manipulated on screen. These elements are eachmapped to respective locations shown at 508′, 510′, and 512′ in thetouch area as defined by touch surface 502. A user may position a fingeror other object at or near 512′ and touch surface 502. This input can berecognized as selection of item 512. Area 512′ may have a distincttexture or other haptic effect associated with the content or identityof item 512 in some embodiments.

To move item 512 to window 510, the user may drag across touch surface502 across borders 514′ and 516′, corresponding to borders of windows508 and 510 in interface 506. If the second location (window 510 in thisexample) corresponds to a protected area (e.g., a system or othersecured folder), an appropriate surface-based haptic effect may beprovided as border 516′ is approached or crossed. As another example, ifelement 512 represents a protected file, haptic effects may be providedat or near border 514′. For instance, the coefficient of friction may beincreased significantly to discourage movement out of window 508 and/orinto window 510. As another example, the status of element 512 may beindicated by a particular texture at area 512′ or a particularly highcoefficient of friction when a movement begins from area 512′ ascompared to movement of other, non-protected elements (not shown).

FIG. 5B illustrates an example where the processor is configured toselect a haptic effect based on a control displayed in the graphicaluser interface at a location mapped to a position in the touch area ator near the position of the touch. Additionally, haptic effects based onadditional content are shown.

Particularly, graphical user interface 520 includes a first portion 522displaying content and a second portion 524 featuring on-screencontrols. In this example, content portion 522 features a map with amajor road 526, minor road 528, and a destination 530. Control portion524 includes a slider 532 and buttons 534 and 536. Any suitablegraphical control elements can be used, including, but not limited to,checkboxes, radio buttons, drop-down menus, accordion menus, dials, andthe like.

Graphical user interface 520 is mapped to the area of touch surface 502.In this example, touch surface 502 is illustrated to provide a number ofsurface-based haptic effects. Although illustrated simultaneously, itwill be understood that the various haptic effects indicated bycross-hatching will be provided by device 501 as a touch is at or nearthe area containing the effect.

As shown at 538, the portion of the touch area mapped to content portion522 features a border 538. The border may comprise a texture or frictionvariation to help indicate the boundaries of the portion mapped tocontent portion 522. Additionally, surface effects are provided at 544and 546 to identify major road 526 and minor road 528.

For example, different types of road and other map features may havedistinct textures or friction levels. As another example, an on-screenroute may be displayed in content portion 522. Portions of touch surface502 mapped to the route may have a different friction or texture thanportions corresponding to roads off of the route. For example, a usermay be able to trace the route by following low-friction portions 544and 546, with friction increasing past location 548. Location 548, asthe intended destination, may have a unique friction or texture.

Touch surface 502 features a portion 550 representing a haptic effectthat may serve to guide a user toward a portion of the touch area mappedto control portion 524. For example, portion 550 may feature a lowercoefficient of friction to allow an easy path to the controls.

As shown at 552A, 552B, and 552C, different haptic effects are providedas the value of slider 532 increases. For example, the friction levelmay change as the slider is moved upward or downward. As anotherexample, buttons 534 and 536 may feature distinct border areas signifiedby different textures or simulated by friction variances. In someembodiments, the texture or friction of buttons 534 and 536 varies basedon a value or state of the button. Haptic effects in addition to textureand/or friction can be provided as well. For example, as a user movesinto/from a button, a haptic pop or vibration may be provided.

Other Illustrative Embodiments of Surface-Based Effects

The examples above were provided for purposes of illustration and arenot intended to be limiting. Additional examples of surface-based hapticeffects are discussed below.

In some embodiments, surface-based effects are provided independently ofa mapping to a graphical user interface. For example, a touch surfacemay be used to provide inputs to manipulate graphical elements in athree-dimensional environment presented in the graphical user interface.Different textures may be provided to indicate different controls ormotions—for example a first texture (or friction) along the x axis inthe touch surface may indicate translation in an x-y plane in the 3-Denvironment, while a second texture (or friction) along the x-axis mayindicate translation in an x-z plane of the 3-D environment. As anotherexample, manipulation of an object in the 3-D environment may beindicated by varying textures or friction along multiple contact pointsof the touch surface.

As a further example, different textures can be used to indicatedifferent type of control actions. For example, in a computer aideddesign (CAD) or 3-D environment, a first texture may be provided whenmoving an element or object in the environment, while a second textureis provided when the view is changed but the element/object is notmoved.

Still further, embodiments include changes in texture associated withdrawing or other input. For example, in a design application, differentbrushes or other tools can have corresponding textures. As a brush sizeis changed, the brush size may be “felt” by providing a differenttexture with each brush size and/or by mapping the brush texture to alarger portion of the touch area.

Embodiments include the use of texture, friction, and/or other hapticeffects in accordance with the present subject matter in order toindicate the contents of a file or document. For example, a userinterface may include a timeline selection mechanism (e.g., a slider)that includes surface-based haptic effects. For instance, an audio filemay be analyzed for one or more characteristics (e.g., volume) andsurface-based haptic effects mapped to the characteristics (e.g.,different textures, friction levels, etc. for different volumes). As theuser interacts with the selection mechanism, different haptic effectscan be provided (e.g., the slider feels different when portions ofincreased volume are encountered).

As another example, surface-based haptic effects may be used inconjunction with games. For example, U.S. Patent Application PublicationNo. 2008/0064499, which is incorporated by reference herein in itsentirety, describes the use of haptic effects with casino and othergames. Surface-based haptic effects may be used in addition to orinstead of effects described therein, such as differentfrictions/textures when playing cards, game instruments (e.g., roulettewheel, dice, etc.) are interacted with via the touch surface and/or toindicate results or game rules (e.g., different texture when a useradvances, different friction values used to indicate when a prohibitedmove is made in a game, etc.).

GENERAL CONSIDERATIONS

The use of “adapted to” or “configured to” herein is meant as open andinclusive language that does not foreclose devices adapted to orconfigured to perform additional tasks or steps. Additionally, the useof “based on” is meant to be open and inclusive, in that a process,step, calculation, or other action “based on” one or more recitedconditions or values may, in practice, be based on additional conditionsor values beyond those recited. Headings, lists, and numbering includedherein are for ease of explanation only and are not meant to belimiting.

Embodiments in accordance with aspects of the present subject matter canbe implemented in digital electronic circuitry, in computer hardware,firmware, software, or in combinations of the preceding. In oneembodiment, a computer may comprise a processor or processors. Theprocessor comprises or has access to a computer-readable medium, such asa random access memory (RAM) coupled to the processor. The processorexecutes computer-executable program instructions stored in memory, suchas executing one or more computer programs including a sensor samplingroutine, a haptic effect selection routine, and suitable programming toproduce signals to generate the selected haptic effects as noted above.

Such processors may comprise a microprocessor, a digital signalprocessor (DSP), an application-specific integrated circuit (ASIC),field programmable gate arrays (FPGAs), and state machines. Suchprocessors may further comprise programmable electronic devices such asPLCs, programmable interrupt controllers (PICs), programmable logicdevices (PLDs), programmable read-only memories (PROMs), electronicallyprogrammable read-only memories (EPROMs or EEPROMs), or other similardevices.

Such processors may comprise, or may be in communication with, media,for example tangible computer-readable media, that may storeinstructions that, when executed by the processor, can cause theprocessor to perform the steps described herein as carried out, orassisted, by a processor. Embodiments of computer-readable media maycomprise, but are not limited to, all electronic, optical, magnetic, orother storage devices capable of providing a processor, such as theprocessor in a web server, with computer-readable instructions. Otherexamples of media comprise, but are not limited to, a floppy disk,CD-ROM, magnetic disk, memory chip, ROM, RAM, ASIC, configuredprocessor, all optical media, all magnetic tape or other magnetic media,or any other medium from which a computer processor can read. Also,various other devices may include computer-readable media, such as arouter, private or public network, or other transmission device. Theprocessor, and the processing, described may be in one or morestructures, and may be dispersed through one or more structures. Theprocessor may comprise code for carrying out one or more of the methods(or parts of methods) described herein.

While the present subject matter has been described in detail withrespect to specific embodiments thereof, it will be appreciated thatthose skilled in the art, upon attaining an understanding of theforegoing may readily produce alterations to, variations of, andequivalents to such embodiments. Accordingly, it should be understoodthat the present disclosure has been presented for purposes of examplerather than limitation, and does not preclude inclusion of suchmodifications, variations and/or additions to the present subject matteras would be readily apparent to one of ordinary skill in the art.

What is claimed is:
 1. A system comprising: a processor in communicationwith a haptic output device, configured to: receive user input on atouch sensitive display configured to display a user interfacecomprising a page; determine a haptic effect to generate based onidentifying a position of a gesture and the page, wherein the hapticeffect is configured to adjust a coefficient of friction of the touchsensitive display to simulate turning the page; and output a hapticsignal associated with the haptic effect to a haptic output deviceconfigured to output the haptic effect while the touch is occurring. 2.The system of claim 1, wherein the haptic output device comprises one ormore of: an ultrasonic actuator or an electrostatic actuator.
 3. Thesystem of claim 1, wherein the gesture comprises input from a right sideof the user interface to a left side of the user interface.
 4. Thesystem of claim 3, wherein adjusting the coefficient of frictioncomprises increasing the coefficient of friction as the gesture movesfrom the right side of the user interface to the left side of the userinterface.
 5. The system of claim 1, wherein the processor is furtherconfigured to determine the haptic effect to vary the coefficient offriction in a direction of the gesture as the gesture is provided. 6.The system of claim 1, wherein the processor is configured to determinethe haptic effect in response to recognizing a gesture indicatingmovement of an on-screen content item from a first location to a secondlocation.
 7. The system of claim 1, wherein an adjustment of thecoefficient of friction of the touch sensitive display comprisesincreasing the coefficient of friction of touch sensitive display. 8.The system of claim 1, wherein the page comprises a plurality of pagesand wherein the haptic effect is configured to simulate turning theplurality of pages.
 9. A method comprising: receiving user input on atouch sensitive display configured to display a user interfacecomprising a page; determining a haptic effect to generate based onidentifying a position of a gesture and the page, wherein the hapticeffect is configured to adjust a coefficient of friction of the touchsensitive display to simulate turning the page; and outputting a hapticsignal associated with the haptic effect to a haptic output deviceconfigured to output the haptic effect while the touch is occurring. 10.The method of claim 9, wherein the haptic output device comprises one ormore of: an ultrasonic actuator or an electrostatic actuator.
 11. Themethod of claim 9, wherein the gesture comprises input from a right sideof the user interface to a left side of the user interface.
 12. Themethod of claim 11, wherein adjusting the coefficient of frictioncomprises increasing the coefficient of friction as the gesture movesfrom the right side of the user interface to the left side of the userinterface.
 13. The method of claim 9, wherein determining the hapticeffect comprises determining the haptic effect to vary the coefficientof friction in a direction of the gesture as the gesture is provided.14. The method of claim 9, wherein determining the haptic effectcomprises determining the haptic effect in response to recognizing agesture indicating movement of an on-screen content item from a firstlocation to a second location.
 15. The method of claim 9, wherein anadjustment of the coefficient of friction of the touch sensitive displaycomprises increasing the coefficient of friction of the touch sensitivedisplay.
 16. The method of claim 9, wherein the page comprises aplurality of pages and wherein the haptic effect is configured tosimulate turning the plurality of pages.
 17. A non-transitory computerreadable medium comprising program code, which when executed by aprocessor is configured to cause the processor to: receive user input ona touch sensitive display configured to display a user interfacecomprising a page; determine a haptic effect to generate based onidentifying a position of a gesture and the page, wherein the hapticeffect is configured to adjust a coefficient of friction of the touchsensitive display to simulate turning the page; and output a hapticsignal associated with the haptic effect to a haptic output deviceconfigured to output the haptic effect while the touch is occurring. 18.The non-transitory computer readable medium of claim 17, wherein thegesture comprises input from a right side of the user interface to aleft side of the user interface.
 19. The non-transitory computerreadable medium of claim 17, wherein adjusting the coefficient offriction comprises increasing the coefficient of friction as the gesturemoves from a right side of the user interface to a left side of the userinterface.
 20. The non-transitory computer readable medium of claim 17,wherein determining the haptic effect comprises determining the hapticeffect to vary the coefficient of friction in a direction of the gestureas the gesture is provided.